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DETECTING A COMPOSITION OF AN AUDIENCE 

¥mJD OF THE DISCLOSURE 
[0001] This disclosure relates generally to audience demographics 

measurement and, mora particularly, to methods and apparatus for detecting a 
composition of an audiance of an information presenting device. 

BACKGROUND 

[0002] Audience measurement of broadcasted television and/or radio 

programs has been practiced for maaiy years. Audience measm:ement devices 
typically collect two lands of information &om households, namely, tuning 
information (e.g., information indicating the content presented to the audience 
such as channel information, time of consumption information, program 
information, etc.) and people information (e.g., information about the 
demographics of the audience). These two types of information are combined 
to produce meaningful ratings data. 

[0003] People information has historically been gathered by people 

meters. People meters have been constructed in many different manners. For 
example, some people meters are active devices which seelc to determine flie 
composition of the audience by, for instance, analyzing visual images of the 
audience to actively determine the identity of the people in the audience. Such 
active determination involves comparing facial features of an individual 
appearing in a captured image to one or more previously stored facial feature 
images to search for a match. Other people meters are passive devices which 
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prompt 1iie members of the viewing audience to identify themselves by 
logging tliemselves in at specific times. These specific prompting times can 
be independent of the tuning information and at fixed time intervals (i.e., time- 
based prompting) or they can be tied to the tuning information and be 
performed, for example, when the channel changes (i.e,, channel change-based 
prompting). 

[0004] The time-based prompthig technique poses a danger of under 

sampling or over sampling the data. For example, if the prompts are spaced 
too far apartin time, audience members may enter or leave the room between 
pron^ts. If the audience does not notify the people meter of such 
entrances/exite, audience composition data and audience change timing is lost. 
Alternatively, if the time prompts are spaced too closely in time, the audience 
members may become annoyed and/or reduce their compliance with the 
prompt requests. Again, audience composition data is lost in such 
circumstances, 

[0005] The channel change-based prompting technique discussed 

above poses the danger of over sampling the data. As explained above, such 
overly frequent prompting may cause irritation and/or result in a decrease in 
comphance and a corresponding loss of data collection and/or mvaHd data. 

BRIEF DESCRIPTION OV THE DRAWTNOR 
[0006] FIG. 1 is a schematic illusteation of an example apparalns 

constructed in accordance with tbe teachings of the invention. 
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[0007] FIG. 2 is a more detailed sclieraatic illusliation of the example 

apparatus of FIG. 1. 

[0008] FIG. 3 is a schematic illustration of an example implementation 

of the apparatus of FIGS. 1-2. 

[0009] FIGS. 4A-4D are a flow chart illustrating example machine 

readable instructions which may be executed by the apparatus of FIG. 3 to 
implement the i^paratus of FIGS. 1-2. 

[00103 FIG. 5 is a schematic illustration of an example people counter 

constructed in accordance witii the teachings of the invention. 
[0011] FIG. 6 is a schematic illustration of an example blob 

discriminator. 

[0012] FIGS. 7A-.7C are a flow chart illustrating example macliine 

readable instructions which may be executed by the apparatus of FIG. 3 to 
implement the apparatus of FIGS, 5 and 6. 

[0013] FIGS. 8A-8G illustrate example histograms developed by the 

apparatus of FIGS. 5 and 6. 

DETAILED DBSrTiTPTTnw 
[0014] FIG. 1 is a schematic Ulustration of an example apparatus 10 

for detecting a composition of an audience of an infomation presenting device 
(not shown). The information presenting device may be. for example, a 
television and the audience may be, for example, a statistically sampled 
household selected to develop television ratings data. Alternatively, the 
information presenting device may be a peisonal video recorder, a computer 
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yone or more 

if one or 



monitor, a radio with or without a visual display, or any other communication 
device designed to present information for consumption by o 
individuals. Similarly, the audience can be made up of any group of oi 
more individuals. For example, ihe group need not be selected via statistical 
sampling or any other technique. In the following, it is &rther assumed that 
demographic information (e.g.. age. sex, ethnic background, income level, 
education level, etc.) concerning each of the expected audience members has 
been collected and stored in association with unique expected audience 
member names orpseudo names in a conventional fashion. As a result, when 
the apparahis 10 obtains the name(s) orpseudo name(s) of the audience 
memb6r(s), it has also effectively obtained Ihe demographic composition of 



[0015] AsshownmFIG. 1, the apparatus 10 includes an audience 

change detector 12 and a content collector 14. The audience change detector 
12 caphires one or more hnages of the audience; determines a number of 
people withm the imageCs); and prompts the audience to identify its members 
if a change in the number of people in the audience is visuaUy detecfEd. The 
content coUector 14 monitors source data to identify a program being 
consumed (e.g., viewed, Ustened to. etc.) by the audience. Persona of ordinary 
skill m the art wiU readily appreciate that any known technique can be utilized 
to identify the program being consumed. For example, the content collector 
14 may identify a consumption time and a source of the program being 
consumed by the audience. Ths consumption time and the source 
identification data may be utilized to identify the program by, for example. 
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cross-referencmg a program guide configured, for example, as a look up tatle. 
The source identification data may, for example, be the identity of a tuned 
channel (e.g., channel 3) obtained, for example, by monitoring the tuner of tlie 
information presenting device. The source data and tiae consumption time 
may be recorded for later use in identifying tiie program either locally or 
remotely following ejcportation of the data, and/or the source data and the 
consumption time may be utilized inunediatsly for on-the-fly program 
identification. 

[0016] Alternatively or additionally, in Ihe visual presentation context 

(e.g., television viewing), codes embedded in the vertical blanldng interval of 
tine program being viewed may be uti'Iized by the content collector 14 to 
positively identify tiie program being consumed by the audience. 
[0017] A detailed aiustration of an example implementation of the 

apparatus 10 is sbown in FIG. 2. As shown in HG. 2, flie audience change 
detector 12 of FIG. 2 includes an image sensor IS to capture images of tiie 
audience consuming the program(s) presented on the information presentation 
device. Images are preferably only captured when the information presenting 
device is in an "on" state. TTie image sensor 1 8 may be implemented in any 
known way. For example, it may be implemented by an infiared imager, or a 
digital camera such as a chaige-coupled device CCD camera. 
[0018] For tiie purpose of determining a number of people appearing 

in the images captured by tiie image sensor 18. tiie audience change detector 
12 of the apparatus 10 is further provided with a people counter 20. Tlie 
people counter 20 may deterrame the number of people wifltin tiie image(s) in 
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many different ways. However, a preferred metliod identifies people within 
tlie imageCs) by detecting changes indicative of movement between successive 
images. An example people counter 20 and an example manner of 
implementing the same are discussed below in connection with FIGS. 5-8. 
[0019] In order to determine if the number of audience members has 

changed, lie audience clrnnge detector 12 is further provided mth a change 
detector 22. The change detector 22 compares the number of people counted 
in the image(s) by the people counter 20 to a value representative of a previous 
number of people in the audience. The value repissentative of the previous 
audience count may, for example, be fte audience count the people counter 20 
developed in analyzing the last image or set of images, or, in, for example, the 
case of the first audience image analysis (e.g., tlie first imageCs) collected after 
apower-up event), it maybe a default value (e.g.. 0). If a difference exists 
between the audience count developed by the people counter 20 and die 
previous number of people in the audience, the change detector 22 develops an 
oulput signal indicating an audience composition change has been detected. 
[0020] As shown inFIG. 2, the audience change detector 12 includes a 

prompter 24 which is responsive to the output signal developed by the change 
detector 22 to request the audience to identify its members. If the change 
detector 22 identifies a difference between die number of people in the 
image(s) and the value representative of the previous number of people in the 
audience, the prompter 24 outputs a signal to the audience prompting the 
audience to identify the individual(s) in the room and/or to identify any 
individual(s) thathave entered or left the room. The signal can be any type of 
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human perceptible signal. For example. Hie prompter 24 may be implemented 
by a visual display and tlie signal output by the prompter 24 may he a 
vie^vable request For instance, the visual display may be the television screen 
or a separate, dedicated display device and the visual signal may be a menu 
requesting the audience to identify the current audience member(s) (or 
alternatively any newly departed and/or newly added audience raember(s)) 
ta a list of predetermined possible members. Alternatively, the prompter 24 
may be a flashing light or an audible sound providing a sensible signal to the 
audience that an ajidience count change has been detected. 
[0021] Regardless of die type of signal employed (e.g., visual, audible, 

etc.), in the illustrated example the people comiter 20, the change detector 22 
and the prompter 24 cooperate to prompt the audience member(s) to log 
themselves(a) in whenever a change in the number of audience members 
occurs. As a result, the audience is neither oversampled (i.e.. prompted 
excessively), nor undersampled (i.e., prompted too infrequently such that 
audience change times are missed). Also, in die event all audience members 
leave the room, the apparatus 10 automatically detects and records that there is 
no audience members, thereby collecting accurate audience measurement data 
even when no audience member is present to respond to a prompt. 
[0022] In order to receive data from the audience member(s), the 

audience change detector 12 is further provided with an input device 26 such 
as a conventional IR transmit-receive pair, a mouse, a keyboard, a 
touchscreen, a touchpad, a microphone and voice recognition engine, and/or 
any other means of inputting date into a computmg device. lulhe example 
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Shown in the figures, the input device 26 is an ffi. leceiver and the audience is 
provided with one or more conventional IR transnutters for remotely entering 
data into ttie apparatus 10. 

[0023] As also shown in FIG. 2, the audience change detector 12 

includes a time stamper 28 and a memory 30. The time stamper 28 includes a 
conventional clock and calendar, and functions to associate a time and date 
with recorded events. For example, if the change detector 22 detects a change 
in tlie audience count, it outputs the counted number of audience members to 
the time stamper 28 and/or the memory 30. The time stamper 28 tlien 
associates a time and date with the new audience count by, for example, 
appendmg the time/date data to the end of the audience count Ihe complete 
data paclcage (i.e.. audience count, time and date) is stored in the memory 30. 
Similarly, whenever data such as, for example, the identity of an audience 
member is entered via the input device 26, flie time stamper 28 associates a 
time and date wifli the data. The memory 30 stores tiie entered data, time and 
date for later analysis. 

[0024] Forthepuipose of determining if one ormoremembers of tlie 

audience is not being identified in response to a prompt from Ihe prompter 24, 
the audience change detector 12 is further provided with a compliance detector 
32. As shown in FIG. 2, Ihe compliance detector 32 mom'tors tlie inputs from 
the audience and compares them to tiie audience count developed by (he 

people counter 20. Ifanumberofmembers identified by the audience via the 
input device 26 is different from flie determined number of people after a 
predetenninednumber of prompts of the audience, the change detector 22 
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causes &s difference between lie number of members identified by flie 
audience and the number of people detennined from flie image(s) by the 
people counter 20 to be recorded in the memory 30 as a number of 
unidentified audience members. The time stamper 28 ensures the records 
indicative of the presence of the unidentified audience memb6r(s) is time 
stamped and dated, as explained above. 

[0025] In the event such unidentified audience member(s) are detected, 

tlie compliance detector 32 adjusts a value representative of the previous 
number of people in flie audience by a difference between the number of 
members identified by the audience and tlie number of people determined 
from the imageCs) by the people counter 20 to avoid excessive prompting of 
the audience. In other words, the value indicative of the last audience comit 
made by the people counter 20 is adjusted so that, assuming the audience 
composition does not change in the interim, at tlie next image collection and 
evaluation by the people counter 20, the change detector 22 wiU compare the 
audience count developed by the people counter 20 to an audience count 
which includes the unidentified audience member(s). Therefore, since inthis 
example, no change in lie number of audience members has occurred, the 
change detector 22 wiU not detect a change and the prompter 24 will not 
prompt the audience even though the unidentified audience memberCs) are 
present As a result, the compliance detector 32 functions to avoid excessively 
prompting the audience if an audience member is reiusing to identify 
himselfiTierself 
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[0026] In flie example of FIG. 2, the content collector 14 includes a 

program detector 34 and an output device 36. The progmm detector 34 
monitors source data to determine flie Bource of the program being consumed 
by the audience. For example, the program detector 34 may monitor the tuner 
of the information presenting device (e.g, a television) to determine which 
chamiel is currently tuned Ce.g., a television program on chamiel 3 is being 
viewed). Alternatively or additionally, the program detector 34 may monitor a 
video screen of the information presenting device to determine if a visual 
somrce identification signal is present during one or more vertical blanldng 
intervals of the program being consmned. Alternatively or additionally, the 
program detector 34 may monitor an audio output of the information 
presenting device to determine if an audio source identification signal is 
present in the program being consumed. Irrespective of how the data is 
gathered, the detected source information is time stamped by the time stamper 
28 and stored in the memors^ 30 for subsequent analysis. 
[0027] In the example of FIG. 2, tlie output device 36 periodically 

exports the recorded data from I3ie memory 30 to a remote data analysis 
computer (not shown) via a network such as the Internet or the like. The data 
analysis computer identifies audiences (e.g., the individualCs) comprising an 
audience and, thus, the demographic composition of the audience) and flie 
programs, or parts of programs, those audiences consumed. This analysis can 
be performed, for example, by cross-referencing the recorded time, date and 
source data for the subject audiences to a program guide. Alternatively, the 
data analysis could be performed locaUy and exported via a network or the 
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Iflce to a data coUection computer for further processing. In either event, the 
data collection computer typically assembles data from multiple different 
households to develop ratings data. No images are transmitted or extracted 
&om the apparatus 10 under either the local analysis or remote analysis model. 
The exportation of data can be done through a wired or wireless connection. 
[0028] An example apparatus 60 for implementing the apparatus 1 0 of 

FIGS. 1-2 is shQ%vn inFIG. 3. The apparatus 10 of FIG, 3 includes an image ' 
sensor 1 8 such as an analog camem and a digitizer 52 for digitizing the analog 
imageCs) captured by the image sensor IS into digital data. The image sensor 
1 8 and digitizer 52 may alternatively he implemented by a single device such 
as a digital camera. 

10029] The apparatus 50 of flie instant example includes a processor 

54. For example, the processor 34 may be implemented by one or more 
Intel® microprocessors from the Pentium® family, the Itanium™ family or 
theXScale^^My. Of course, other processors from other families are also 
appropriate. 

r0030] As is conventional, the processor 54 is in communication with 

a main memory 30 via a bus. memory 30 stores Ibe data developed by the 
apparatus 10. R also stores computer readable instructions which, when 
executed, cause the processor 54 to determine a number of people within lie 

imageCs) captured by the sensorlS, and to developapromptsignal requesting 
the audience to identify its member(s) if a change in the nmnber of people in 
the audience is visually detected 
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[0031] The memory 30 may include a volatile memory and a non- 

volatile memory. The volatile memory may be implemented by Synclironous 
Dynamic Random Access Memory (SDRAM), Dynamic Random Access 
Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM) 
and/or any other type of random access memory device. The non-volatile 
memory may be implemented by flash memory and/or any other desired type 
of memory device. Access to the main memory 30 may be controlled by a 
memory controller (not shown) in a conventional manner. 
[0032] The memory 30 may also include one or more mass storage 

devices for storing software and data. Examples of such mass storage devices 
mclude floppy disk drives, hard drive disks, compact disk drives and digital 
versatile disk (DVD) drives. 

[0033] The apparatus 50 also includes a communication block or 

interface circuit 56. The interface circuit 56 may be implemented by any type 
of well Icnown interface standard, such as an Ethernet interface, a universal 
serial bus (USB), and/or a thud generation input/output (3GI0) interface. 
[0034] One or more input devices 56 are included in or connected to 

the interface circuit 56. The input dBvice{s) permit a user to enter data and 
commands into the processor 54. The input device{s) can be implemented by, 
for example, an IR transmit/receive pair, a Iceyboard, a mouse, a touchscreen, 
a track-pad, a trackball, isopoint and/or a voice recognition system. 
[0035] An output device 24 is also corniected to the interface circuit 

56. The output device 24 is responsive to the prompt signal output by the 
processor 54 to output an indication requesting the audience to identify its 



-12- 



wo 2004/054255 



PCT/US2002/039fi25 



members. In the example of PIG. 3, the output device 24 is a liquid crystal 
display (LCD) which outputs a visually perceptible prompt signal. However, 
the output device 24 may additionally or alternatively be implemented by, for 
example, other visual and/or audible display devices (e.g., a cathode ray tube 
(CRT) display, a printer and/or speakers). 

[0036] The interface circuit 56 also includes a coinmunication device 

such as a modem or network interface card to fecilitate exchange of data with 
external computers via a network (e.g., an Ethernet connection, a digital 
subscriber line (DSL), a telephone line, coaxial cable, a cellular telephone 
system, etc.). It may also include a communication device such as an mfrared 
decoder to receive and decode IR signals transmitted to the apparatus 60 by 
one or more audience members 

[0037] An example software program for implementing the apparatus 

of PIGS. 1-2. is shown in FIGS. 4A-4D. In this example, the program is for 
execution by a processor such as the processor 54 shown in the example of 
FIG, 3, and the program is embodied in software stored on a tangible medium 
such as a compact disk (CD), a floppy disk, a hard drive, a digital versatile 
disk (DVD), or a memory associated with tlie processor 54. However, persons 
of ordinary sldll in the art will readily appreciate that tlie entire program or 
parts thereof could alternatively be executed by a device other than the 
processor 54 and/or embodied in firmware or dedicated hardware in a well 
known manner. For example, any or all of the people counter 20, the change 
detector 22, the time stamper 28. the program detector 34, and/or the 
compliance detector 32 could be implemented by software, hardware, and/or 
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finnware. Further, although the example program is described with reference 
to tlie flowchart illustrated in HGS. 4A-4D, persons of ordinary sldll in fee art 
will readily appreciate that many other methods of implementing the example 
apparatus 10 may alternatively be used. For example, the order of execution 
of I3ie bloolcs may be changed, and/or some of the blodcs described may be 
changed, eliminated, and/or combined. 

[0038] In ftie example of HG. 4A, tlie program starts at power up 

when the processor 54 engages in various conventional housekeephig tasks 
such as initializmg Ihe memory, etc. {block 100). TTie people counter 20 then 
resets the variables LAST COUNT and CURRENT COUNT to zero (block 
102). The processor 54 then determined whether the information presenting 
device (in this example, a viewing device such as a television) is in an on state 
(block 104). If the mformation presenting device is in an off state, the 
processor 54 enters a wait loop until the information presenting device is 
turned on. No audience images are coUected unless the mfoimalion presenting 
device is in an "on" state. 

[0039] Assuming the information presenting device is in an on state, 

tiie people prompter 24 is driven to prompt the audience to identify its 
member(s) {block 106). The apparatus 60 then awaits an input &om the 
audience (block lOS). When a:i audience input is received via, for example, 
the input device 26 (block 108), the processor 54 updates the database mit the 
input data (e.g., an audience member's identify) (block 1 10). The time 
Stamper 28 may record a time and date in association with the input data. The 



-14- 



wo 2004/054255 



PCT/US2002/039625 



people counter 20 then increments the LAST COUNT variable to reflect the 
presence of the audience member that identified himselfiTierself (block 1 12). 
[0040] The processor 54 then detennines if a predetermined length of 

time Ce.g., 10 seconds) has elapsed since the last input was received from the 
audience (block 1 14). If the predetermined time has not elapsed (block 1 14). 
control returns to block 108 where lie processor 54 determines if anotlier 
audience input has been received. If not. control again proceeds to block 114. 
Otherwise control advances to block 110 where the database stored in the 
memory 30 is updated to reflect the mw audience data input Control 
continues to loop through blocks lOS-1 14 until no audience inputs are 

received for the predetermined length oftime(bIockll4). at which pointit is 
assmned that all audience members have identified themselves (although this 
assumption is tested at block 146 as explained below). 
10041] Assuming tire predetermined length of time has elapsed witiiout 

any furflaer audience inputs (block 1 14), contol proceeds to block 116. At 
block 1 16. the program detector 34 identifies tlie source of the program being 
presented on the information presenting device. If a change m the source has 
occurred (e.g.. tuning changed from channel 3 to chamiel 1 1). or if a power on 
event just occurred (e.g., tuning changed &om no tuned cbamiel to chamiel 12) 
(block 118). the database stored in the memory 30 is updated ™fli the new 
source information (block 120). As explained above, tiae time stamper 28 
associates a time and date with the new source informatioa If no source 
change ortiam on event has occurred (block IIS). control skips block 120 and 
proceeds directly to block 122. 
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E0042I At block 122, the image sensor 18 is activated to capture 

iniage(s) of the audience. The captured imageCs) are digitized (block 124) and 
passed to the people counter 20. The people counter 20 ihsn analyzes the 
imageCs) to determine if any person is located in the imageCs) as explained 
below in connection will FIGS. 5-8 (block 126). The variable CURRENT 
COUNT is set to reflect the number of persons in the image(s) (block 128). 
Control then advances to block 134. 

[0043] At block 134, the change detector 22 determines if lie 

CURRENT COUNT value (i.e.. flie number of persons counted in the captured 
image(s)) is equal to the LAST COUNT value (i.e., the number of persons 
counted inmiediately prior to the capturing of tlie image(s) being analyzed). If 
the CURRENT COUNT value and the LAST COUNT value are equal (block 
134), control returns to block 116 because no audience change has occuixed, 
Otherwise, conti-ol proceeds to block 136. Control continues to loop through 
bloclcs 116 -134 until an audience count change is detected (block 134). 
[0044] Assuming an audience count change has been detected (block 

134), the time stamper 2S updates the database in tlie memory 30 ™fh an enhy 
indicating the time and date that an audience change occurred (block 136). It 
then drives lie prompter 24 to prompt tlie audience member(s) to identify 
themselvesCs) (block 138, FIG. AQ. If an andience input is received (block 
140), the new data is written to the memory 30 (block 142). If no audience 
input is received for a predetermined length of time (e.g.. 10 seconds) (block 
144), control advances to block 146. Otherwise control continues to loop 
through blocks 140-144 as long as the audience continues to input data. 
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[004S] Assuming tbat lite audience has stopped inputting data (block 

144), the compliance detector 32 determines if the number of audience 
members identified by the inputs received firom the audience is equal to the 
CUREENT COUNT developed firom the captured image(s) by tlie people 
counter 20 (block 146). If the audience identified less audience members than 
the people counter 20, then tlie compliance detector 32 determines whether 
this discrepancy has occurred a predetermined number of times sequentially 
(e.g., tliree times in a row) (block 148). Ifnot, control proceeds to block 150. 
Otherwise, control advances to block 156 of FIG. 4D, 
[0046] Assuming for the moment that the number of audience 

members identified in the inputs received from the audience is equal to the 
number of individuals counted by tlie people counter 20, the compliance 
detector 32 sets the lAST COUNT variable equal to the CURRENT COUNT 
value (block 150). Setting the LAST COUNT variable in this mamier ensures 
that only changes in lie audience count result in audience prompts (see block 
134). After the LAST COUNT variable is set (block 150), the CURRENT 
COUNT variable and the NONCOMPLLWT PERSON COUNT variable are 
both re-set to zero (block 152). 

[0047] The program detector 34 then verifies that the mformation 

presenting device is still in an on state (block 154). If so, control returns to 
block 1 16 (FIG. 4B). Control contiiiues to loop tlirough bloclcs 116-154 untU 
the information presenting device is turned off (block 154). or noncompliant 
persons are detected a predetetmined number of times sequentiaUy (block 
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148). If the Morniation presenting device is turned ofif (block 154), control 
returns to block 102 (FIG. 4A). 

[0048] Assuming that at least one audience member refiises to identify 

himselfOierself for 1he predetermined number of times sequenliaUy (block 
148), control advances to block 156 (FIG. 4D). At block 156. the compUance 
detector 32 calculates the number of unidentified individuals in the audience. 
In particular, the noncompliance detector 32 sets tiie NONCOMPLIANT 
PERSON COUNT variable equal to the value in llie CURRENT COUNT 
variable minus the number of audience members identified by the audience 
inputs. TheNONCOMPLIANTPERSONCOUNTisthenwrittentothe 
memory 30 in association witiia time and date stamp, thereby recording the 
number of unidentified persons in tiie audience (block 158). The variable 
LAST COUNT is then inciBmented by the value in the NONCOMPLIANT 
PERSON COUNT variable (block 160). Adjustmg the LAST COUNT 
variable m Aiis manner avoids repeatedly prompting tire audience to identify 
noncompliant persons. 

100491 After re-setting the CURRENT COUNT variable to zero (block 

1 62). tie program detector 34 then verifies fliat Hie information presenting 
device is still in an on state (block 164). If so, control returns to block 116 
(FIG. 4B). on the other hand, Hie information presenting device is turned 
off (block 164), control returns to block 102 (FIG. 4A). 
[0050] An example people counter 20 is schematicaUy illustralEd in 

FIG. 5. The people counter 20 shown in FIG. 5 includes a motion detector 
1 SO. The motion detector 180 receives a sequence of images fiom the image 



-18- 



wo 2004/054255 



PCT/US2002/(139fi25 



sensor 18. In the example of FIG. 5. the image sensor IS captures and 
provides images to the motion detector ISO at a rate of IS fiamea per si 
although other rates of capture would Ukewise be appropriate. 
10051] The sequence of images may be digitized when the motion 

detector 180 receives them, or, alternatively, the motion detector 180 may 
include a digitizer 52 to convert the images received &om the image sensor 18 
hito digital images. In the example of FIG. 5. the images are digitized such 
that each pixel in each image is assigned an 8 bit binary value (i.e., a value of 
0 - 255) which is representative of the corresponding image data. Thus, each 
image can be thought of as an array of digital data with each element 
contained in tiie array corresponding to an 8 bit binary value. Hie number of 
pixels assigned to each image (i.e., the resohition) can be selected at any 
desired level. However, the same array size is preferably employed for each 
i^nage. AddilionaUy. JPEG (or other picture format) copies of the origmal 
images may be saved for fiitnre reference, if desired. 
[0052] The motion detector 180 operates on each sequential pair of 

images received &om the image sensor 18 to detect motion occurring between 
tlie two inmges. More specifically, assuming a given room containing an 
audience is repeatedly photographed to create a series of images as explained 
above, then if there is no movement for a given time period, there will be no 
significant difference between two successive images of the room taken during 
the period of no movement. Thus, tlie binary values of the elements in the 
image array for a first image wiU be identical (or substantially identical if 
noise enrors or the like are present) to the binary values of tie coiresponding 
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elements in the image anay for a second image taken immediately after flie 
first image. If, however, there is movement between the time at which the 
first image is token and the time at which the second image is talcen, tlie binary 
values of tlie elements in the image array for the second image will be 
different from the bmary values of the corresponding elements in the image 
array for the second image. 

[0053] The morion detector ISO takes advantage of this fact to detect 

differences due to, for example, motion of audience members between 
successive images received from the hnage sensor 18 by comparing each 
successivepahrofimagesonanelementbyelementbasis. hi particular, the 
motion detector ISO develops a difference image correspondmg to each pair of 
successively received images by subtractmg the coaesponding elemente of 
oneimageairay&omtheolherimagearray. In an extremely simplified 

exampIewhereineachdigitizedimagaisanarmyoffburelements,assmning 
that the elements of a first received image have the following values (90, 1 03, 
23, and 203), and tlie coirespondmg elements of a second received image have 
the foUowing values (90. 1 03, 60 and 250). then the difference image 
computed by the motion detector is an array of four elements having the 

following values (0,0,-37.-47). In flns example, there has been motion 
between the first and second image and. thus, some of the values in the 
difi-erence image are non-zero. The non-zero values represent pomts of 
motion. If there is no difference between successive images, all the values in 
the difference image corresponding to those two successive images will be 
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zero (or substantially zero as some smaU differences may appear due to noise 
or other error). 

[0054] From the foregoing, persons of ordinary skill in the art wiU 

appreciate that each difference image is typically a collection of motion points 
localized around center(s) of motion. In order to correlate these motion points 
to ohjects m the images, the people counter 20 is forther provided with a shape 
outliner 182. The shape outliner 182 employs a process such as the well 
known convex hull algorithm to draw shapes or blobs encompassing the 
motion points. As is well known by persons of ordinary skill in the art, the 
convex hull algorithm joins all points in a set of pomts that satisfy a 
predetermined constraint into a blob or shape. The predetermined constraint 
may be a requirement feat aU of the points in llie blob or shape are separated 
by less ftan a predetermined distance. Since m this example, we are 
attempting to identify humans. Uie predetennined distance should be a 
distance correspondingto the size of a human being. This distance may be a 
settable or programmable parameter and may be set based on the sizes of the 
expected audience members at a given household, 
E00S5] Since there may not be enough data points in a difference 

image for the shape outliner 1 82 to draw meaning&l shapes, the people 
comiter 180 is fiirfher provided with an image amalgamator 184. For each 
image for which the people counter 20 is requested to develop a people comit, 
the image amalgamator 1S4 integrates or otherwise smoothes or filters the 
difference images from a time interval in which fte image to be analyzed is 
located into a single amalgamated image. For example, if the image to be 
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analyzed OGoura at time i, the image amalgamator 184 will combine liie 
difference images ftom a time interval Ijeginning at time i-k and ending at 
time i+c into a smgle image array, where Ic and c are preferably equal, but may 
be different The difference images may be combined into an amalgamated 
image by summmg the airay oorrespondmg to the difference images on an 
element by element basis and then dividing each summed element by the 
number of elements smnmed (i.e., the number of difference images). Thus, 
hke the arrays conesponding to the difference images, the amalgamated image 
is an amy of 8 bit bmary values (i.e., values ranging from 0 to 255). 
[0056] As shown in FIG. 5. rather than acting directly on Hie 

difference images, the shape outliner 1 82 operates on the amalgamated image 
corresponding to an image being analyzed to draw blob(s) within the 
amalgamated hnage via die process explamed above. Operating on the 
amalgamated images rather than direcHy on the difference images integrates or 
averages error over a fixed time interval, which has a tendency to reduce the 
size of noise objects that could be interpreted as motion relative to objects that 
are representative of actual motion. 

[0057] From the foregoing, persons of ordinary skill in the art will 

appreciate that the motion detector 1 80, the image amalgamator 1 84 and the 
shape outliner 182 function to reduce flie problem of comiting people 
appearing in an image to counting bIob(s) reflecting centerCs) of motion within 
an image. 

[0058] For the purpose of discrhninating human blob(s) appearing 

within the amalgamated image from non-human blobCs) (e.g., pets, random 
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noise, manimate olDjeots, eta.), the people counter 20 may optionally be furiher 
proidded with a non-human filter 188. In the iHuslrated example, the non- 
human filter 188 analyzes the ahape(s) drawn within the amalgamated image 
by the shape outliner 182 to determine if any can be eliminated from Hie 
amalgamated image as not possibly corresponding to a human being. The 
non-human filter 188 may employ any logical test to eliminate blob(s) firom 
the amalgamated image. For example, the non-human filter 188 may test the 
location(s) of He bloh(s) to determine if their looation(s) identify them as not 
human. For instance, a blob located on the ceiling of a room can be 
eliminated as not humaa In addition to location based tests, the non-human 
filter ISS may also testflio size of the shape. For example, if the size of abioh 
is beneath a certain threshold or above a certain threshold, it may be 
eliminated as not reflecting a human sized object T]ie tests performed by the 
non-hmnan filter 1 S 8 may be adjusted to suit the household being analyzed. 
For example, in a household with children, the non-human filter 188 may 
employ a lower size threshold flian a household with no children. Similarly, in 
a household with no children, the non-human filter 188 may identify blob(s) 
appearing on 1i,e floor as non-hmnan. whereas is may not be allowed to 
identify blob(s) on Ibe floor as non-human based purety on a floor location if 
tlie household includes children. If the test(s) employed by the non-human 
filter 1 88 are to be tailored to the demographics of the household being 
analyzed. Hie testCs) should be adjusted at set up of tlie apparatus 20. 
[0059] The non-human filter 188 may eliminate a blob ftom the 

amalgamated image in many different ways. For example, the binary values 
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in the amalgamated image giving rise to the object being eliminated can be 
zeroed, and the revised amalgamated image fed back to the shape outliner 1 82 
to create a new set of blob(s) in the amalgamated image excluding Ihe blob(s) 
eliminated by 1he non-human filter 1 S8, 

[0060] For file purpose of detemiining if any of ihe blobCs) appearing 

in the amalgamated image (optionaUy, as filtered by flie non-hnman filter 188) 
represents a person, tlie people counter 20 is fiirther provided with a blob 
discriminator 190. Were one to simply comat liie number of blobs appearing 
in the amalgamated image (oplionany as filtered by the non-human filter 188), 
false people counts might result in certain mstances. For example, if two 
people are located in an audience, but only one of those people moves during a 
time period being analyzed, only one blob will appear m the amalgamated 
hnage. and shnply counting blobs without felher refinement would result in 
an undemount By way of another example, if two audience members move m 
a symmetrical fashion for a given period of time, liey could potentiaHy appear 
as a smgle blob in the amalgamated image. Simply counting blobs in this 
scenario will again result m an undeicount. The blob discriminator 190 solves 
this potential problem by ensuring only blob(s) that exhibit persistent motion 
over a time period of mterest are counted as persons. 
[0061] Toperibrrathepersistentmotiontest. theblob discriminator 

190 does not develop a count of the blobs appearing in every amalgamated 
image, fcstead. a number of sequential amalgamated images are analyzed 

overaperiod of thne.ln particular, for each amalgamated image, the blobCs) 
contained therem are represented by symbols in a histogram. Although a blob 
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can appear only once in any given amalgamated image, if the blob exhibits 
persistent motion, it will appear in multiple different amalgamated images. 
For every time a blab appears in an amalgamated image and meets the convex 
huU criteria, a symbol is added to the histogram. Tlierefore, the histogram 
tracks the number of times each blob exhibits motion over a period of time. 
After that period of time, tlie histogram is analyzed and only iJiose blobs that 
have exhibited sufficient persistence of molion as indicated by tiie number of 
times a symbol corresponding to that blob appears in the histogram, are 
identified as persons, 

[0062] An example blob discriminator 190 is shown in FIG. 6. For tlie 

purpose of identifying the center of gravity of the blob(s) appearing in the 
amalgamated image, the blob discriminator 190 is provided with a center 
locator 192. In the illuslxated example, the center locator 192 computes lie 
center of gravity of each blob in the amalgamated image by assigning a value 
to a plurality of points m the blob. The value assigned to a given point 
corresponds to the X-axis location of the given point in Uie amalgamated 
image. The X-axis may, for example, coriBspond to the field of view of the 
image sensor. Once these point values are assigned, the center locator 192 
averages the values. The average X-axis value computed by the center locator 
192 corresponds to the X-axis position of the center of gravity of the blob in 
question. The computed center of gravity is then added to flie histogram 
which is used to test the persistence of the motion of each identified blob as 
explained above. 
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10063] In order to add a symbol which is representative of the center of 

gravity of the blob to the histogram, Iha blob discriminator 190 is further 
provided witii a center comparator 194. The center comparator 194 serves a 
gravitation fiinction. In particular, whenever the center locator 192 computes 
a center of gravity of a blob, the center comparator 194 compares the newly 
computed center of gravity to the existing centers of gravity already appearing 
in the histogram. If the newly computed center of gravity is the same as, or 
falls within a predetennined distance of, a center of gravity already 
represented in the liistogram, it is assumed that the newly computed center 
represents the same object as the existing center. As a result, a symbol 
representative of flie newly computed center is added to the symbol 
representing the existing center in the histogram. Preferably, every symbol 
added to the histogram has the same size. Therefore, when a symbol is added 
to one ormore existing symbols in the histogram, the existing symbol "grows" 
in size. 

[0064] Persons of ordinary skill m tlie art will readily appreciate that a 

histogram such as that described above may be hnplemented in many different 
ways. For example, it may be implemented graphically wherein symbol(s) of 
the same size are placed at the X-axis location of their corresponding blob(s). 
If two ormore symbols have substantiaUy the same X-axis location (thereby 
exhibiting some level of persistent motion of their corresponding object), they 
are stacked verticaUy. Ahematively. a horizontal gro^vth metric may be used. 
Alternatively or additionaUy, the histogram could be implemented by a set of 
cowiters wherem each counter in the set corresponds to an X-axis location 
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within an amalganiated image. If a blob having a center of gravity 
conesponding to the X-axis location of a given counter is identified in an 
amalgamated image, the concesponding counter is incremented. Therefore, the 
larger the number of times a blob appears in a series of amalgamated images, 
the Imger the value m the coiresponding counter becomes. 
[0065] To determine whether any symbol in the liistogram has 

exhibited sufficient persistent motion to be counted as a person in the 
audience, the blob discriminator 190 is furflier provided with a threshold 
counter 198. Tlie tlireshold counter 19S compares the number of times each 
center of gravity is represented in the histogram to a predetermined threshold. 
This can be done, for example, by comparing the size of the symbol to the 
predetennined tlureshold. If any symbol in the histogram has a size greater 
than the threshold, it is counted as a persoa Thus, m the example of HOS. 1- 
4, the CURRENT COUNT variable is mcremented one time for every symbol 
having a size that exceeds the predetermined ftreshold. 
[0066] In the people counter 20 of FIGS. 5-6, a histogram and, thus, a 

people count, is not developed foi every possible sequence of amalgamated 
unages. Instead, a histogram is made only when there is suiBcient motion in 
the room being monitored to suggest tliat an audience composition change 
may be occurring (e.g., someone walking into or out of a room appears as a 
large amount of motion compared to an audience sitting in front of an 
information device). To detennine when to develop and analyze a histogram, 
the blob discriminator 190 is further provided with an energy detector 200. 
For each difference unage developed by the motion detector 180, the energy 
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detector 200 computes an energy value. In tlie illustrated example, the energy 
value is computed by squaring each anray representing a difference image and 
summing liie values corresponding to iie elements in the squared array. If the 
summed value exceeds a predetermined energy threshold, tiie difference image 
has a corresponding energy level Ihat suggests an audience change may be 
occurring. Therefore, whenever a difference image that exceeds the energy 
threshold is detected by the energy detector 200, the energy detector 200 sets a 
motion marker in association wifli that difference image. 
[0067] The people counts developed at tl:e motion markers can be 

ejctxapolated to the periods occurring between motion markers. This 
extrapolation is possible because there is relatively Uttle motion between the 
motion mari£ers. Someone entering or leaving the audience room is associated 
with a significant amount of motion. Since motion markers are set when such 
a significant amomit of motion occurs, no significant amount of motion occurs 
between motion markers, and it can, thus, be safely assumed that no one has 
left or entered lie room in the time between motion maricers. Therefore, it can 
be safely assmned that the audience composition has not changed in the time 
between motion markers. By way of aoi example, if at a first motion marker 
the people counter 20 determines there are 2 people in the audience, and at tlie 
next motion marlcer the people counter determines tiiere are three people in tiie 
room, tlien, because no motion indicating a person has entered or exited the 
room is detected prior to tiie second motion marker, the people count for the 
entire period ftom the first motion marker to the second motion marker is two 
people. 
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[OOeaj When sufficient data has Lesn developed around a motion 

marker (e.g., when enough amalgamated images prior to and after a motion 
marker have been analyzed for the corresponding histogram developed from 
those amalgamated images to have meaning, the threshold counter 198 is 
activated. As explained above, the threshold counter 198 deteimines whether 
any blob represented in the histogram has exhibited sufficient persistence of 
motion to be counted as a person. Any such blob is counted as a person and 
the person count so developed is output by tlie people counter 20. 
10069] To prevent noise and false motions fi^m cluttering the 

hislDgram, the blob discriminator 190 is further provided with a false motion 
filter 202. The false motion detector 202 of the illustrated example reviews 
the symbols recorded in lie histogram as the histogram is being developed. If 
any symbol does not grow for a predetermined amount of time (e.g., tliree 
minutes), the symbol is assumed to be noise or other false motion and is 
eliminated firom the histogram. Li tliis way, erroneous entries due to noise, 
random movement, or enroneous consohdation of two or more blobs into one 
blob aiB not allowed to grow. 

[0070] An example software program for implementing liie apparatus 

20 of FIGS. 5-6. is shown in FIGS. 7A-7C. In this example, the program is 
for execution by a processor such as the processor 54 shown in the example of 
FIG. 3, and the program is embodied in software stored on a tangible medium 
such as a compact disk (CD), a floppy disk, a hard drive, a digital versatile 
disk p VD), or a memory associated with the processor 54. However, persons 
of ordinary skill in Hie art will readily appreciate that the entire program or 
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parts thereof could alternatively be executed by a device other than the 
processor 54 and/or embodied in finaware or dedicated hardware in a well 
know manner. For example, any or all of the motion detector 180, the shape 
outliner 182, the image amalgamator 184, the blob discriminator 190, the non- 
human filter 188, the center locator 192, the center comparator 194, the energy 
detector 200, the threshold counter 1 98 and/or the false motion filter 202 could 
be implemented by software, hardware, and/or firmware. Firrther, although 
the example program is described with reference to the flowchart iUustrated in 
FIGS. 7A-7C, persons of ordinary sldll in the art will readily appreciate that 
many other methods of implementing the exEimple people counter 20 may 
alternatively be used. For example, the order of execution of the bloclcs may 
be changed, and/or some of the bloclcs described may be changed, eliminated, 
and/or combined. 

[0071] The program of FIGS. 7A-7C may be used witli the program of 

FIGS. 4A-4D to implement the apparatus 10. If so implemented, the program 
of FIGS. 7A-7C replaces bloclcs 122-128 of FIG. 4B. However, persons of 
ordinary skill in the ait wiU readily appreciate Hiatthe program of FIGS. 7A- 
7C could be implemented wiffiout the program of FIGS. 4A-4D or vice versa. 
For example, the program of FIGS. 4A-4D could use an entirely different 
method of counting people in an image and/or the program of FIGS. 7A-7C 
could be used for applications other than audience measurement. In the 
following, it is assumed that Ihe program of FIGS. 7A-7C is used in the 
program of FIGS. 4A-4D. Therefore, in this example, control enters the 
program of FIG. 7A via block 118 or 120 of FIG. 4B. 
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[0072] Turning to FIG. 7 A, the program begins when the image sensor 

1 S captures an image of the audience in question (block 220). The digitizer 52 
then digitizes the captured image into an array of eight bit values as explained 
above (block 222). If this is the first image captured by the sensor 18 (i.e., the 
program has just been started after a power-off tune), then two images are 
captured and digitized (blocks 220-222). Hie digitized image(s) aie then 
saved m the memory 30 (block 224). 

[0073] The motion detector 1 80 then computes the difference image 

between lie most recently captured image and the immediately proceeding 
image stored in memory (block 226). As discussed above, the difference 
image is calculated by subtracting the elements of the mostrecently captured 
image array from the corresponding elements of the most recently stored 
image array in accordance with tiie conventional rules of linear algebra. 
[0074] Once the difference image is calculated (block 226). the energy 

detector 200 cabulates tlie energy value associated with tlie difference image 
(block 228). As explamed above, this energy value is computed by squaring 
the array of the difference image and then summing all of the values contained 
in flae array generated by tlae squaring operation. The energy value is stored in 
memory 30 for later use as explained below. 

[0075} Because many of the calculations performed by the people 

counter 20 require data corresponding to images taken before and after a 
motion marker, it is necessary to have a runnmg sequence of pictures to 
operate upon. IHerefore, before creating any amalgamated unages, the people 
counter 20 creates a buffer of captured images and difference images. Thus, at 
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block 230, the people counter 20 detemiines whether the desired buffer has 
been created. If not, control loops back to block 220 via block 232. At block 
232 a captured image counter i is incremented. Control continues to loop 
through blocks 220-232 until the desired buffer of captured images and 
difference images corresponding to those captured images has been created 
(block 230). 

[0076] Assuming the desired buffer is in place (block 230), control 

advances to block234. At block 234 a number of counters are initialized. For 
example, an amalgamation comiter A is set to equal the image counter i less 
the buffer size. A histogram loop counter B is set to equal the amalgamation 
counter less a delay value sufficient to ensure all needed amalgamation image 
arrays have been computed prior to initiating population of a histogram. A 
range counter K is set to equal the amalgamation counter A - Z. a variable 
settmg the earliest difference image to be used in creating an amalgamated 
image corresponding to time A. A range counter M is set to equal the 
histogram loop counter E - P. a variable settmg the earhest amalgamated 

image to be nsedmcreatingahistogramcon^pondingtoamotion marker 
occuningatthneB. A threshold T is set equal to the amalgamation counter A 
+ E, a variable setting Ihe latest difference image to be used in creating an 
amalgamated image corresponding to time A A second threshold U is set to 
equal the histogram loop counter B + F, a variable settingtiie latest 
amalgamated image to be used in creating a Mstogram correspondmg to a 
motionmarteroccurringattimeB. AdditionaUy, the amalgamation array Sa 
for timeAis cleared to an empty set. Persons of ordinary skiUmthe art will 
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appreciate that the variables Z and E may optionaUy be identical. Similarly, 
the variables P and F may optionally be identical. 

[0077] Once the variables are initialized as explained above (block 

234), the image amalgamator 1 84 sums the amalgamation array SA with die 
difference image anay associated with time K on an element by element basis 
(block 236). The counter K is then incremented (block 23S). If the counter K 
has not surpassed the threshold T (block 240), control returns to block 236 
where die image amalgamator 1 S4 adds the next difference image array to the 
amalgamated rniage array. Control continues to loop through blocks 236-240 
until the counter K equals or exceeds the threshold T (block 240). 
[0078] When tlie compilation of fte amalgamated image array is 

completed (block 240), the image amalgamator 1 84 converts Uie amalgamated 
image airay SA into abinary image (block242). Convertmg the amalgamated 
image array to a binary image can be accomplished by, for example, dividing 
each element in the amalgamated image array by the number of difference 
images used to form the amalgamated image army (e.g., by (Z + E)). 
[0079] The energy detector 200 ten detemines whether the energy 

value associated with time B is greater than an energy threshold X (i.e.. 
whether a motion marlcer is set at time B) (block 244). The energy threshold 
X is a value that indicates the amount of movement energy that is required in a 
difference image to suggest timt mx audience composition change is occurring. 
If a motion marker is set at time B. then control advances to block 250. 
Otherwise, the people counter routine detemiines whedier it has been 
executing for too longof atime (block 246). If so, the people counter routine 
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terminates and control advances to iDlook 134 of FTG. 4B. As with many 
bloclcs of FIGS. 7A-7C, block 246 is not required in aU applications. It is 
included in the illustrated example wherein the people counter xoutine is 
inserted in the program of FIGS. to enable the program to periodically 

check for source changes (block 1 1 8). Were blocks 246 not employed in this 
context, it would be possible to miss several source changes occurring while 
the audience composition remains constant as control woiild othenvise 
continue to loop within the people counter routine until a motion marker were 
reached. 

[0080] Assuming it is not thne to exit the people counter routine to 

check for source changes or a turn-off event (block 246). control advances to 
block 248. At block 248 the captured image counter i is incremented. Control 
then returns to block 220 (FIG. 7A) where anolher image is captured. Conliol 
continues to loop through blocks 220-248 until a motion marker is reached 
(block 244), or a time to check for a source change or tum-off event is reached 
(block 246). 

[0081] Assuming for purposes of discussion fliat a motion maflcer is 

located at time B (block 244), control enters a loop wherem a histogram 
corresponding to the time period begimmig at time M (i.e., time (B -P)) and 
ending at time U (i.e., time (B + F)) is populated. Tn particular, at block 250, 
the shape outliner 1 82 executes tiie convex huU process on the points 
, "appearing in the amalgamated image array SMcon^sponding to time M. As 
explamed above, if any points are presenting the amalgamated image array 



-34- 



wo 2004/054255 



PCT/US2002/039625 



. SM. the execution of the convex hull process draws one or mom blob(s) in Ihe 
amalgamated image array SM. 

[0082] Once tlie blob(s) (if any) are drawn, lie non-human filler 188 

perffanns one ormore logic tcst(s) onUie blob(s) to attempt to eliminate non- 
human blob(s) from the amalgamated image anay SM (block 252). As 
explained above, many different logic teste may be used for this purpose 
including, by way of examples, not limitations, a location test and/or a size 
test. 

[0083] Oncethe non-human filter 188 has completed execution, the 

center locator 192 calculates 1he center of gravity of each remaining blob (if 
any) in the amalgamated image array SM (block 254). As explained above, 

this calculation may be perfonned by averaging theX-axis values for each 
point in the blob in question. 

[0084] Irrespective of how the center(s) of tiie blob(s) are identified, 

once the centers are calculated, the center comparator 192 attempts to record 
the blob(s) in the histogram. In particular, die center comparator 192 
determines if flie center of a first one of tiie blob(s) (if any) in tiie 
amalgamated image SM is located within a predetennined distance Y of a 

center of an object already recorded in the histogram (block 256). The 
predetermined distance is preferably selected to correspond to ti.e expected 
size Of a person along the x-axis of an image (e.g., 40 pixels). As explained 
above, this test is perfonned to ensure that slight differences in tiie centers of 
blobs do not cause the same blob to be identified as two different blobs m 
different amalgamated images. If the center of the blob mider consideration is 
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within Y distance of a center akeady ejdsting in the histogram (block 256), a 
symbol representative of the blob under consideration is added to the symbol 
represantmg the already existing center in. the liistogram (block 258, FIG. 7C), 
If the center of the blob under consideration is not within Y distance of a 
center already existing in flie histogram (block 256). a symbol representative 
of the blob under consideration is added to the histogram as a new center 
representing a new blob (block 260, FIG, 7C). 

[0085] Irrespeotive of whetlier control passes through block 258 or 

260, when control reaches block 262, the center comparator 194 determines if 
there are more blobs to analyze within the amalgamated image SM under 
examination. If so, control returns to block 256 (FIG. 7B). Control continues 
to loop arough blocks 256 - 262 until every blob appearing in the 
amalgamated image SM has been represented in tlie histogram. Control then 
advances to block 264. 

[0086] At block 264, the range comiter M is incremented. The blob 

discriminator 196 then determines whether the loop counter M is equal to or 
greater than fhe threshold U (block 266). If not, then all of the amalgamated 
images to be represented in the histogram have not yet been analyzed, and 
control advances to block 268. Oflierwise. the histogram is complete and 
control advances to block 272. 

10087] Assuming for purposes of discussion that the histogram is not 

yet fiilly populated (block 266), the false motion filter 202 examines Hie 
histogram to determine if any symbols in the histogram have failed to grow 
within a predetermined time period (e.g., 3 minutes)(block 268). If any such 



-36- 



wo 2004/054255 



PCT/US2002/039625 



inactive symbols exist {block 268). the false motion filter 202 assumes these 
inactive symbols are not representative of people and removes them ftom the 
histogram (block 270), Control then returns to block 250 CFIG. 7B) wherein 
the shape outliner 1 82 draws blob(s) around any points present in. tlie next 
amalgamated image SM, If no inactive symbols exist in the histogram Oalock 
268), control advances directly ftom block 268 (FIG. 7C) to block 250 CFIG. 
7B). 

[0088] Control continues to loop through blocks 250 - 270 until tlie 

loop counter M becomes equal to or greater than the tlureshold U (block 256). 
The liistogram is then complete and ready for analysis, Accordingly, tlie 
histogram is latched and stored. 

[0089] Tlie threshold counter 198 then begins analyzmg each symbol 

representative of a blob center appearing in the histogram (block 272). If a 
symbol being exammed exceeds a predetermined threshold (block 272), flie 
thresbold counter 198 identifies the symbol as representative of a person. 
Accordingly, the CURRENT COtMT variable is incremented (block 274). If 
the symbol being examined does not exceed the predetermined flireshold 
(block 272), the threshold counter 198 concludes that the symbol represents 
something other than a person and flie CUEJRENT COUNT variable is, 
therefore, not incremented (block 272). The threshold counter 198 tlien 
determines if every symbol in the histogram has been analyzed (block 276). If 
not, control returns to block 272. Control continues to loop through bloclcs 
272 -276 until every symbol in flie histogram has been identified as human or 
non-human and the human symbols have been counted (block 276). Once this 
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process is completed (block 276), the histogram is cleared for the next round 
of analysis (block 27S). The people counter routine then terminates. In the 
example of FIGS. 4A-4D, control then advances to block 134 of FIG. 4B. 
[0090] To provide further illusfiation of the operation of the people 

counter 20 discussed in conneclion with FIGS. 5-7, an example histogram 
which could be generated by such an apparahis is shown in FIGS. 8A-SG. In 
the example of FIG. 8A, two blobs are present m the first amalgamated image 
examined by tlie blob discriminator 190. The centers of fliese blobs ate 
calculated by the center locator 192 as bemg separated by more than the 
distance Y (e.g., 40 pixels). Accordingly, the two blobs are represented by 
two separate symbols A and B. Each of the symbols is located atHio X-axis 
location of the blob it represents. 

[0091] As shown in FIG. 8B, the next amalgamated image examined 

by the blob discriminator 190 contains 4 blobs. A first blob has a center 
identical to the center of symbol A. Therefore, a symbol representing the first 
blob is stacked on top of the symbol A such that symbol A "grows." 
Similarly, a second blob having a center identical to the center of symbol B is 
present in the second amalgamated image. Accordingly, symbol B also grows 
by virtue of the addition of another symbol to its height. TTie remaining two 
blobs have calculated centers that are separated by a distance greater dian Y 
from both the center represented by symbol A and the center represented by 
symbol B. Accordingly, two new symbols C and D are added to the histogram 
at X-axis locations corresponding to tiie centers of tlie blobs they represent. 
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[0092] The fljird amalgamated image contains only one blob. As 

shown m PIG. 8C, Hiat blob has a center located less than the distance Y from 
the center represented by symbol A (see symbol E). Accordingly, Ibe center 
comparator 194 assmnes that symbol E and symbol A represent the same 
object, and as shown in FIG. SD, symbol E is merged with symbol A such that 
symbol A again grows. 

[0093] HG. SE represents the state of the histogram after several 

additional amalgamated images have been analyzed. As shown in FIG. SE, 
the symbols A. B and C have aU grown (although at different rates) smce tlie 
time reflected inHG 8D. Symbol D. however, has not grown in that time 
period. Accordingly, as shown inHG. 8F, the false motioa filter 202 assmnes 
the inactive symbol D conresponds to noise or some other non-human source, 
and the symbol D is, therefore, eliminated from the histogram. 
[0094] As also shown inFIG. 8F, the amalgamated image being added 

at this time has two blabs. One blob has the same center as symbol A and is, 
thus, merged with that symbol (compare FIG. SE). The other blob (see 
symbol F) has a sligWiy different center than symbol B. However, the center 
of the second blob is less than the distance Y from the center represented by 
symbol B. Accordmgly, as shown in FIG. 8G. the symbol F is merged with 
the symbol B. 

[0095] In the final state of the histogram reflected in FIG. 8G, symbols 

A and B have again grown and a new symbol G conesponding to a third blob 
appearing in the last amalgamated image has been added. In the final latohed 
state shown inHG. 8G, the symbols A and B ore seen to have grown beyond 
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the tiireshold ftequency. Therefore, when the tlireshold counter 198 examines 
file histogram, it identifies sjmbols A and B as corresponding to persons, but 
symbols C and G are considered to be representative of non-hmnans. 
Accordingly, the threshold counter 19S counts two persons in the example of 
FIG. SG. 

[0096] Although certain example methods and apparatus constructed 

in accordance with the teachings of the invention have been described herein, 
the scope of coverage of this patent is not limited thereto. On the conlraiy, 
this patent covers all embodiments of the teachings of the invention fairly 
falling within the scope of the appended claims either literally or under the 
doctrina of equivalents. 
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WhatlsClaimert U - 

1 . A metbod of detecting a composition of an audience of an 

information presenting device comprising: 

capturing at least one image of the audience; 

detemiining a numljer of people within the at least one image; and 

prompting the audience to identify its members if a chmge in the 

nmnber of people is detected based on the number of people determined to 

be within the at least one image. 

2. A method as defined in claim 1 wherein capturing the at least 
one image of the audience comprises digitizing fhe at least one image. 

3 . A metliod as defined in claim 1 wherein determining the 
number of people within the at least one image comprises: 

(a) detennining at least one difference image based on at least two 
captured images; 

(b) developing at least one.shape from the at least one difference 
image; 

(c) identifying at least one center of the at least one shape; 

(d) adding a symbol having a predeiermined size and indicative of the 
at least one center of fhe at least one shape in a histogram; 

(e) repeating (a)-(d); and 

(f) if any symbol in the histogram grows beyond a predetermined 
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tiireshold, counting that symbol as a person. 



4. A method as defined in claim 3 wherein determining the at 
least one difference image based on the at least two of the captured 
comprises: 

(a) determining a first difference image between a first image and a 
second image; and 

(b) determining a second difference image between the second image 
and a third image. 

5. A method as defined in claim 4 further comprising developing 
an amalgamated image fem Ihe first and second difference images. 

6. A method as defined in claim 5 wherein developing the at least 
one shape fi:om the at least one difference image comprises developing Hie 
at least one shape fi:om the amalgamated image. 

7. A method as defined in claim 3 wherein identifying the at least 
one center of tiie at least one shape comprises identifymg a center of a first 
shape and a canter of a second shape. 

8. A method as defined in claim 7 wherein adding the symbol 
having the predeteimined size and indicative of tlie at least one center of 
the at least one shape to the histogram comprises adding a first symbol 



-42- 



wo 2004/054255 



PCT/US2002/039625 



indicative of the center of the first shape to the histogram and adding a 
second symbol indicative of the center of liie second shape to the 



9. A method as defined in claim 8 wherein repeating (a)-Cd) 
comprises stacking a third symbol on the first symbol if a second difference 
image contains a shape having a center tlrnt substantiaUy corresponds to the 
center of the first shape. 

10. A method as defined in claim 3 further comprising excluding a 
shape ftom a group of possible human shapes based on a test. 

11. A mefliod as defined in claim 10 wherein the test comprises at 
least one of a location test and a size test. 

12. A method as defined in claim 3 fiirther comprising identifying 
an energy value associated with the at least one difference image, and, 
perfoiming (f) if the energy value exceeds a predeteimmed threshold. 

13. A method as defined m claim 3 wherem developmg the at least 
one shape fi-om the at least one difference image comprises executmg a 
convex hull process. 
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14. A method as defined in claim 3 whsrein, if any symbol in the 
histogram does not grow within a predetermined length of time, that 
symbol is eliminated fiom the histogram. 



1 5. A method as defined in claim 1 further comprising: 
if a number of members identified by the audience is different from the 
determined nmnber of people after a predetermined number of prompts of 
the audience, adjusting a value to avoid excessive prompting of the 
audience. 



16. A method as defined in claim 15 wherein adjusting the value 
comprises increasing a previous audience count by a difference between the 
number of members identified by the audience and tiae number of people 
determined from the at least one image. 



17. A method as defined in claim 1 6 further comprising recording 
file difference between the nmnber of members identified by the audience 
and the number of people determined firom flie at least one image as a 
number of unidentified audience members. 



18, A method as defined in claim 1 fiirllier comprising periodically 
exporting recorded data to a device to identify audiences and programs 
being consumed by tlie audiences, 
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19. A mefliod as defined in claim 1 further comprising identifying a 
program teing consumed by tlie audience. 

20. A method as defined in claim 19 wherein identifying a program 
being consumed by fhe audience further comprises; 

identifying a consumption time; and 

identifying a source of the program being consumed by fhe audience. 



21. A method as defined in claim 20 fiirflier 
the source and tlie consumption time. 



22. A mGtliod as defined in claim 20 wherein identifying the sourc< 
of the program comprises identifying a tuned channel. 

23. A method as defined in claim 20 fnrflier comprising using tlie 
consumption time and the source to identify tlie program from a program 
guide. 

24. An apparatus to detect a composition of an audience of an 
information presenting device comprising: 

a to capture at least one image of fhe audience; 



a memmy storing computer readable instructions which, when 
executed, cause the processor to: 



-45- 



wo 2004/054255 



PCT/US2002/03!J625 



determine a number of people within the at least one image; 

develop a prompt signal requesting the audience to identify ils 
s if a change in tlie number of people is visually detected; and 
an oulput device responsive to the prompt signal to output an 
indication requesting the audience to identify its members. 

25. An apparatus as defined in claim 24 wherein the oulput device 
is a visual display and the prompt signal is a visual signal. 

26. An apparatus as defined in claim 24 wherein the camera 
digitizes the at least one image. 

27. An apparatus as defined in claim 24 wherein the processor 
deteimines tlie number of people within tlie at least one image by; 

(a) determining at least one difference image between at least two 
images captured by the camera; 

Cb) developing at least one shape from the at least one difference 
image; 

(c) identifying at least one center of Ihe at least one shape; 

(d) adding a symbol having a predetermined size and indicative of the 
at least one center of the at least one shape to a histogram; 

(e) repeating {a)-(d); and 

(f) if any symbol in die histogram grows beyond a predetemiined 
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threshold, counting the symbol as a person, 

28. An apparatus as defined in claim 24 wherein, if a number of 
members identified by the audience is different fiom the detennined 
number of people after a predetermined number of prompts of the audience, 
Hie computer readable instnactions cause the processor to adjust a value to 
avoid excessive prompting of the audience. 

29. An apparatus as defined in claim 28 wherein the processor 
adjusts the value by increasing a previous audience count by a difference 
between fho number of members identified by the audience and the number 
of people determined from the at least one image. 

30. An apparatus as defined in claim 29 wherein the computer 
readable instructions cause the processor to record tiie difference between 
the number of members identified by the audience and the number of 
people determined firom the at least one image as a number of unidentified 



31 . An apparatus as defined in claim 24 wherem the computer 
readable mstructions cause fte processor to periodicaUy export recorded 
data to a device to identify audiences and programs being consumed by the 
audiences. 
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32. An appaiatus as defined in claim 24 wherein the computer 
readable instructions cause the processor to identify the program being 
consumed by the audience. 

33. An appaiatos as defined in claim 32 wherein identifying a 
program being consumed by tlie audience further comprises: 

identifying a consumption time; and 

identifying a source of the program being consumed by Uie audience. 

34. An apparatus as defined in claim 33 wherein the computer 
readable instructions cause the processor to record the source and the 
consumption time. 

35. An apparatus as defined in claim 33 wherein identifymg the 
source of the program fiirther comprises identifying a tuned channel. 

36. An apparatus as defined in claim 33 further comprising using 
the consumption time and the source to identify the program firom a 
program guide. 

37. A machine readable medium storing madiine readable 
instructions which, when executed, cause at least one machine to: 

caphjre at least one image of an audience consuming a program; 
determine a nnmber of people within the at least one image; and 
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output a prompt signal requesting flie audience to identify its members 
if a change in the number of people is detected fcom the at least one image. 

38. A machine readable medium as defined in claim 37 wherein the 
at least one machine determines the number of people within the at least 
one image by: 

(a) determining at least one difference image between at least two 
captured images; 

(b) developing at least one shape from the at least one difference 
image; 

(c) identifying at least one center of the at least one shape; 

(d) adding a symbol having a predetermined size and indicative of tlie 
at least one center of the at least one shape to a histogram; 

(e) repeating (a)-(d); and 

(f) if any symbol in the histogram grows beyond a predetermined 
threshold, counting the symbol as a person. 

39. A machine readable medium as defined in claim 37 wherein, if 
a number of members identiBed by the audience is different &om the 
determined number of people after a predetermined number of prompts of 
the audience, the machine readable instructions cause tiie at least one 
machine to adjust a value to avoid excessive prompting of the audience. 
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40. A maclUne readable medium as defined in claim 39 wherein the 
at least one machine adjusts the value by increasing a previous audience 
couirtby a difference between the number of members identified by tlie 
audience and the number of people determined from the at least one image. 

41 . A machine readable medium as defined in claun 40 wherem the 
machme readable mstniotions cause the at least one machine to record tlie 
difference between the number of members identified by tlie audience and 
the number of people determined &om the at least one image as a number 
of unidentified audience members. 

42. A machme readable medium as defined in claim 37 wherem the 
machme readable instructions cause the at least one machme to periodically 
export recorded data to a device to identify audiences and programs being 
consumed by the audiences. 

43. A machine readable medium as defined in claim 37 wherein the 
machme readable mstmctions cause the at least one maclnne to identify the 
program being consumed by the audience. 

44. A machine readable medium as defined in claim 43 wherein 
identifying a program being consumed by the audience further comprises: 

identifying a consumption time; and 
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identifying a source of the program being consumed by liie audience. 

45. A macbine readable medium as defined in claim 44 wherein the 
machine readable instructions cause the at least one machine to recoi-dthe 
souioe and tlie consumption time. 

46. A machine readable medium as defined in claim 44 wherein 
identifying the source of the program comprises identifying a tuned 
channeL 

47. A computer readable medium as defined in claim 44 further 
comprising using the consumption time and the source to identify the 
program from a program guide. 

48. An apparatus to detect a composition of an audience of an 
information presenting device comprising: 

an image sensor to capture at least one image of the audience; 
ft people counter to detennine a number of people in the at least one 
image; 

a change detector to compare the number of people in the at least one 
image to a value representative of a previous number of people in the 

audience; 

a prompter to request liie audience to identify its members if the 
change detector identifies a difference between the number of people in the 
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at least one image and the value representative of tiie previous number of 

people in the audience; and 

an input device to receive data from the audience. 

49. An apparatus as defined in claim 48 fiirther comprising a 
compliance detector to determine if a member of the audience is not being 
identified in response to the prompt. 

50. An apparatus as defined in claim 48 wherem the prompter is a 
visual display. 

51 . An apparatus as defined in claim 48 wherein the image sensor 
is a digital camera. 

52. An apparatus as defined in claim 48 furfibier comprising a 
digitizer to digitize the image captured by the image sensor. 

53. An apparatus as defined in claim 48 wherein the people counter 
comprises: 

a motion detector to compare at least two images to detect motion; 

an image amalgamator to develop an amalgamated image from the at 
least two images; 

a shape outHner to draw at least one shape within the amalgamated 
image; and 
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a blob discriminator to detemiine if the at least one shape represents a 
person. 

54. An apparatus as defined in claim 53 further comprising a non- 
human filter to eliminate a non-human shape ftom the at least one shape. 

55. An apparatus as defined in claim 54 wherein the non-human 
filter eliminates the non-human shape based on at least one of a location of 
the non-human shape and a size of the non-human shape. 

56. An apparatus as defined in claim 53 wherem the blob 
discriminator comprises: 

a center locator to identify a center of the at least one shape; 
a center comparator to add a symbol representative of the center of the 
at least one shape to a histogram; and 

a threshold counter to count symbols in the histogram exceeding a 
predetermined threshold. 

57. An apparatus as defined in claim 56 wherein if the center of the 
at least one shape substantially corresponds to an existing center in the 
histogram, tJie center comparator adds the symbol representative of tlie 
center of the at least one shape to a symbol representing the existing center 
in the histogram. 
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58. An apparatus as dpjSned in claim 56 further comprising an 
energy detector to compare a value indicative of motion occurring between 
tlie two images to an energy thresliold, and to cause the threshold counter to 
count the symbols in the histogram exceeding the predetermined toeshold 
if the value exceeds the energy threshold. 

59. An apparatus as defined in claim 56 fiirther comprising a false 
motion detector to elhninate a non-growing symbol from the histogram. 

60. An apparatus as defined in claim 48 wherein, if a number of 
members identified by the audience is different from the nmnber of people 
determined by tiie change detector after a predetermined number of 
prompts of the audience, a compliance detector adjusts the value 
representative of the previous number of people in the audience to avoid 
excessive prompting of the audience. 



61. An apparatus as defined in claim 60 wherein the compliance 
detector adjusts the value by increasing the value representative of the 
previous number of people in the audience by a difference between the 
nmnber of members identified by the audience and the number of people 
determined from the image by the people counter. 

62. An apparatus as defined in claim 61 wherein Uie compUance 
detector records the difference between the nmnber of members identified 
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by tiie audience and the number of people determined fiom the at least one 
image as a number of unidentified audience member. 

63. An apparatus as defined in claim 48 fiirther comprising a 
content coUector to identify the program being consumed by the audience. 

64. An apparatus as defined in claim 48 further comprising an 
output device to export recorded data to a device to identify audiences and 
programs bemg consumed by the audiences. 

65. An apparatus as defined in claim 63 wherein the content 
collector further comprises a program detector to identify a source of the 
program bemg consumed by the audience. 

66. An apparatus as defined in claim 65 fiirther comprismg a time 
stamper to identify a consumption time, and a memory to record the source 
and the oonsumption time. 

67. An apparatus as defined in claim 65 wherein die program 
detector identifies the source of ae program by identifyingatunod channel. 
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